• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    A method for capturing images from a camera (12) of a portable device (11) of an audiovisual screen (13) in which a multimedia content of interest for a user of the portable device (11) is played; comprises the steps of capturing, by means of at least one camera (12), at least one source image (14) of the audiovisual screen (13); storage, in a first repository of the portable device (11), of the plurality of source images (14); transmission, through a first bidirectional communications interface, of the source images (14) to an application server; receiving, by means of a second bidirectional communication interface included in the application server, of the source images (14); sending the received images to a processing unit connectable to a second repository, which stores a set of multimedia contents where each stored multimedia content has at least one complementary multimedia content associated; carrying out a matching search, by means of the processing unit, between the set of received images and the multimedia contents stored in the second repository; provides a link to complementary multimedia content associated with the stored multimedia content, by means of the processing unit, in case the result of the search procedure is affirmative; transmission of the link provided from the second communication interface to the first communication interface; and providing the link to the user of the portable device (11) for viewing. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2684690A1
    申请号:ES201700449
    申请日:2017-03-30
    公开日:2018-10-04
    发明作者:Felipe SEGURA GUTIERREZ；Rubén MARTÍNEZ SANDOVAL；Antonio ROBLES PASQUÍN
    申请人:Felipe SEGURA GUTIERREZ；Rubén MARTÍNEZ SANDOVAL；Antonio ROBLES PASQUÍN；
    IPC主号:
    专利说明:

    DESCRIPTION

    Method for capturing images from a portable device.

    Object 5

    The present invention relates to a method for capturing images from a camera of a portable device that allows the creation of a direct communication bridge between any screen that projects multimedia content and any camera that captures multimedia content in a context of everyday reality. 10

    State of the art

    Any multimedia reproduction such as a movie contains information that may be of interest to the viewer. In most cases, the viewer must rely exclusively on the information displayed and only in exceptions does the multimedia content have references such as hashtags, which allow access to additional content to that shown on the screen.

    Today, the number of viewers who use an audiovisual screen of a portable device 20 with a camera to display multimedia content increases very rapidly.

    Document US2012008821 describes a system for capturing multimedia content, visualized by means of an audiovisual screen, from a camera of a mobile phone. This method is based on the detection of the differences in brightness between the screen, in which the content is broadcast, and the environment surrounding the same audiovisual screen; that is to say, in measuring the light contrast between the audiovisual screen and the peripheral region thereof. If the level of contrast is not very pronounced, the system is not able to differentiate multimedia content from the environment of the playback screen.
     30
    Summary

    The present invention seeks to solve one or more of the drawbacks set forth above by a method for capturing images from a camera of a portable device as defined in the claims. 35

    The method comprises the steps of focusing the camera of a portable device towards an audiovisual screen in which multimedia content of interest to a user of the portable device is being reproduced; capturing at least one source image of the multimedia content reproduced from the camera of the portable device, the capture of the source images is carried out continuously or discontinuously; that is, the source images are spaced in time at regular or irregular intervals; The set of source images are stored in a first storage repository of the portable device in order to be transmitted through a first two-way communication interface from the portable device to an application server, which receives the sequence of origin images. captured by means of a second bi-directional communication interface; the source images received are sent to a processing unit that is connected to a second repository which stores a set of multimedia contents where each stored multimedia content has at least one complementary multimedia content associated; the processing unit is configured to carry out a method 50 for matching between the set of received images and multimedia contents stored in the second repository; when the result of the execution of the search procedure is positive; that is, the set of images received on the application server is coincident with images of a stored multimedia content, the unit
    processor provides a link to a complementary multimedia content associated with the stored multimedia content; so that the link supplied by the processing unit is transmitted from the application server, via the second communication interface, to the first communication interface of the portable device. Consequently, the complementary multimedia content is made available to the user of the device for display, for example.

    The complementary multimedia content is associated with the multimedia content reproduced on the audiovisual screen of interest to the user of the portable device.
     10
    A stored multimedia content may have associated a plurality of complementary multimedia contents associated with each one at a time of different reproduction of the stored multimedia content. For example, every 5 seconds of playback of a stored multimedia content has a different complementary multimedia content associated with it. fifteen

    The audiovisual screen on which multimedia content of interest to a user of the portable device is being played may be the screen of a laptop, a television or the like. The portable device can be a device such as a laptop, a netbook, a Tablet, a Smartphone or the like. twenty

    The application server runs the match search program between a request message, which includes a set of captured images, and multimedia content stored in the second repository, and in case of finding a match provides a link to a complementary multimedia content. In the procedure of capturing 25 source images, the portable device camera captures source images that include multimedia content reproduced by the audiovisual screen and the surrounding external environment of the same audiovisual screen.

    The portable device captures and transmits a sequence of source images spaced 30 regularly or irregularly to the application server.

    Once the source images have been received by the application server, the processing unit performs a step of resizing and aligning the received images, which are scaled to a reduced image size to decrease the computation time of the server's processing unit. of application.

    Next, the processing unit aligns the previously resized images with each other.
     40
    Once the received images have been resized and aligned, the processing unit executes a luminosity analysis stage to determine the brightness level of the aligned images, in order to determine the light conditions of the aligned image in order to determine which portion of the image corresponds to the audiovisual screen and which portion of the image corresponds to the surrounding environment to the same screen, at the time of the capture of the source images.

    Consequently, at this stage the portion of the image corresponding to the screen is identified from among all the elements that make up the aligned image; namely, region of interest, ROI. The image portion other than the region of interest corresponds to the potentially non-screen region 50, RNP.

    The processing unit determines the region of interest ROI by detecting sudden changes in the level of RGB brightness present in each aligned image. If the evaluation
    of the brightness, RGB level, in the potentially non-RNP region of the images is homogeneous and is below a predetermined first ThresholdRGB threshold, the region of interest ROI will be delimited by the geometry marked by the strong contrast existing in each of the aligned images.
     5
    If the result of the previous procedure is not satisfactory; that is, it is not possible to define the region of interest from the level of luminosity; the processing unit executes a motion detection procedure, for which the processing unit identifies which points of the aligned image have been moved in response to a vibration of the portable device at the time of the capture of the source image, for example, in front at points 10 the aligned image has been moved by changes between the sequence of aligned images; that is, due to natural changes present in multimedia content. The result of this procedure is to be able to determine the region of interest ROI, which will correspond to the screen portion studying the motion analysis of the image characteristics.
     fifteen
    Once the region of interest of the images aligned by the motion detection procedure has been delimited, the processing unit executes a procedure for detecting possible frames where the quadrilateral that optimizes the surface occupied by the screen is determined by geometric study of the picture.
     twenty
    Once the procedure for the detection of possible frames is finished, the processing unit executes a homogenization procedure whose objective is to complement the conclusions drawn from the procedure for the detection of possible frames, providing a reassessment of all the regions of interest obtained by the procedure of detection of possible frames. 25

    Finally, the processing unit executes a unification procedure that combines the results of the previous procedures and that results in the region of objective interest; that is, it defines a unique solution among all the possible regions of ROI interest. 30

    Once the region of interest ROI has been established, the processing unit extracts from the source images captured with the portable device that region corresponding to the ROI, to execute the procedure of finding a match between the previous images and the multimedia contents stored in the second repository. 35

    The match search procedure is based on an image recognition method within a database using an image detection architecture composed of the sum of the SURF ("Speeded-Up Robust Features"), BOWV ("Bags" of Visual Words ”) and complemented with the results offered by tf-idf (“ Term frequency - 40 Inverse document frequency ”).

    The result of the search is affirmative when there is in the second repository a multimedia content associated with the content of the region of interest delimited above, the associated multimedia content having a complementary multimedia content that can be provided to the user.

    This match search procedure includes three stages:

    SURF stage: allows to describe an image in numerical terms and allows to locate within 50 each image relevant target points such as edges, projections or corners.

    Once the relevant points have been found; namely, characteristic points of the image proceed to describe them mathematically. This description is independent of certain
    transformations such as distortions caused by changes in the ratio of the screens or the smoothing that occurs in the source image when the capture has been done remotely.

    The characteristic points found are treated according to their distance to the center of the ROI region of interest. All the characteristic points within the region of interest are taken in their entirety. The rest of the characteristic points, which are in a perimeter region of the region of interest, are discarded as they approach the perimeter.

    BOVW stage: When the image has been described and characterized by the SURF algorithm, the characteristics generated by the SURF stage enter a processing phase based on a BOVW algorithm, which allows grouping the characteristic points of the SURF algorithm into a word histogram visual Normally, histograms are compared by a measure of dissimilarity that is usually implemented in terms of a mathematical distance such as Euclidean distance, cosine distance or the like. In this stage the comparison between both histograms also considers the frequency of appearance of similar visual words, improving the results before possible distortions in the taking of images.

    For this, a readjustment of the frequencies of occurrence of each of the visual words 20 in the histogram obtained in the BOWV method is necessary. That is, once the BOVW histogram is computed, each value of the latter, which represents the frequency of occurrence of the corresponding visual word, will be reweighted according to the following formula:

    BWWdNABOVWBOVWijijiji)) (( 25

    Where BOVWi is the value of the visual word i (ith value of the histogram), N (∙) represents the value of the Gaussian probability function of mean 0 and variance, d (Wi, Wj) represents the distance, for example , for example Euclidean, between the visual words i-th and j-th, and | ∙ | It is the L1 standard. 30 

    Terms A and B are weight terms that adjust the impact similar visual words have on re-evaluation. A + B is always 1. Normal values of A and B could be A = 0.8 and B = 0.2.
     35
    Thus, the reweighting considers that the values in the BOVW histogram of the visual words similar to the word i - understanding similar as those that have a smaller distance - must modify the value in the BOVW histogram of the term i.

    The search procedure in the second repository avoids redundancies. That is, if the 40 differences between aligned images are reduced, an image is compared with the multimedia contents of the second repository. For example, in a static scene of multimedia content, images equal to each other are analyzed. If not only those images that provide useful information.
     Four. Five
    Stage tf-idf: the application of tf-idf allows you to reevaluate how relevant a visual word is within the set of words considered by the BOVW method. Thus, certain words provide more information than others, due to their global rarity, and should be taken into account more. The tf-idf stage will be applied before the BOVW histogram reweighting stage. fifty

    At the end of the BOVW stage, an image characterized by a histogram created from the descriptors thereof is provided. These histograms represent the frequency with which visual vocabulary words appear in the image under analysis. Thus, the way to find an aligned image in the second repository is carried out by comparing the histogram of the analyzed image with the histograms included within the second repository.

    If the result of the comparison is affirmative, the application server provides a link to a multimedia content complementary to the user of the portable device.
     10
    This link can also provide multimedia content, an opening order for a mobile application, GPS coordinates, a hyperlink or a purchase order.

    In addition, the complementary multimedia content can be edited before being provided to the user depending on the user's location, the profile of the user or according to the timing of the broadcast of the source multimedia content. Therefore, the same stored multimedia content can give rise to a plurality of different complementary multimedia contents.

    Finally, this method identifies partial regions within the delimited region of interest 20 which allows multimedia content to be recognized without the need to delimit the region of interest ROI with complete precision.

    Brief description of the drawings
     25
    A more detailed explanation is given in the description that follows and which is based on the attached figures.

    Figure 1 shows in a diagram the capture of images from a camera of a portable device. 30

    Figure 2 shows in a diagram a plurality of regions of interest for a sequence of two source images.

    Figure 3 shows in a scheme of the region of interest ROI the arrangement of a potentially RPP screen region 35 and the potentially non-RNP screen region.

    Figure 4 represents the regions of interest ROI for a sequence of source images captured by the portable device camera.
     40
    Figure 5 shows the logical flow in block diagram of the path followed in the procedure of sampling, resizing, alignment and analysis of luminosity.

    Figure 6 shows the logical flow in block diagram of the path followed in the first phase of the motion detection procedure.

    Figure 7 shows in a diagram the division of the region of interest ROI into four quadrants.
    Figure 8 shows the logical flow in block diagram of the path followed in the second phase of the motion detection procedure.

    Figure 9 shows the logical flow in block diagram of the path followed in the procedure for detecting possible frames.
    Figure 10 shows the logical flow in block diagram of the path followed in the homogenization procedure.

    Figure 11 shows the logical flow in block diagram of the path followed in the unification procedure. 5

    Figure 12 shows for the coincidence search procedure the treatment of the characteristic points of the image as a function of its distance from the center of the same.

    Figure 13 represents an example of application of the present invention. 10

    Description

    In relation to Figure 1, a user visualizes multimedia content on an audiovisual screen 13 and makes the decision to capture at least one source image 14 of the screen 13 with the camera 12 of a portable device 11.

    The sequence of captured source images 14 are stored in a first repository of the portable device 11, which comprises a first two-way communication interface by means of which the sequence of source images 14 are transmitted to an application server 15 that receives the images captured by means of a second communication interface.

    The application server 15 comprises a processing unit connected to the second bi-directional communication interface and to a second repository which stores multimedia contents and complementary multimedia contents associated with the stored multimedia contents.

    The multimedia content has been stored using its visual word histogram, so that the source images 14 can be compared with the multimedia content 30 stored in the repository of the application server 15 but cannot be viewed through a display screen 13.

    The processing unit executes a procedure for finding a match between the received images and the multimedia contents stored in the second repository; if the search result is affirmative; that is, there is a match, the processing unit provides a link to a complementary multimedia content related to the multimedia content coinciding with the received images. The application server 15 transmits the link provided to the portable device 11 via the second communication interface. 40

    The user can access the complementary multimedia content through the link received on the portable device 11. The complementary multimedia content is associated with the multimedia content displayed on the audiovisual screen 13.
     Four. Five
    In relation to Figures 2 and 3, the processing unit of the application server determines the region of interest ROI, shown in the figure as a striped region, for each image aligned on the application server 15. Each of the captured images The camera 12 of the portable device 11 includes the audiovisual screen 13 and other elements arranged in the area where the screen 13 is located. 50

    The region of interest of the image refers to the portion of the image relative to the screen 13. In the example shown in Figure 2, two aligned images I1, I2 relative to images
    Origin are analyzed to extract their respective areas of interest ROI, RO2 respectively.

    The region of interest ROI for each aligned image is divided into a potentially non-RNP screen region and a potentially RPP screen region, where the potentially RPP screen region is that area of the aligned image within which the audiovisual screen 13 is securely located. . The potentially non-RNP screen region is that area of the aligned image that does not contemplate any screen part safely.

    This consideration takes into account the fact that since the user is the one who orients the portable device 10 towards the screen 13, it cannot be guaranteed that the sequence of captured images 14 fits perfectly to the extent of the screen 13.

    In relation to Figure 4 where the source images 14 represents a sequence of images captured by the camera 12 of the device 11. In the nomenclature used, the sequence of source images is numbered as F0, F1, F2 ... Fn, Fn + one. F0 being the first source image of the sequence captured by portable device 11.

    In figure 4 for each aligned image the respective region of interest ROI has already been extracted; by setting for the first zero aligned image F0 the region of interest ROI1, for the 20 aligned image one F1 the region of interest ROI2, respectively, and so on for the complete sequence of aligned images.

    In relation now to Figure 5, a partial functional block diagram shows the execution of the procedure for sampling, resizing, alignment and luminosity analysis where the study of the external light conditions, which surround the area, is required. audiovisual screen 13.

    At first, the source images 14 that are being captured by the device 11 are scaled to a smaller size. This resizing is done to reduce the computation time of the server processing unit 15.

    Second, images that have already been resized are aligned. This allows to achieve similar geometric conditions between images to improve the results. The resizing work is performed image by image, sequentially, as the source images are captured by device 11. In the alignment step, also sequentially, each Fi image is aligned with the first of the sequence F0.

    Once the images have been resized and aligned, the brightness level of the potentially non-RNP region is analyzed in order to determine the 40 light conditions surrounding the audiovisual screen 13, which shows the multimedia content of interest to the user .

    The device 11 will be considered to be pointing to the screen 13 in night or dark conditions when the brightness level in the region potentially not RNP screen no 45 exceeds a predetermined first ThresholdRGB threshold value; the ThresholdRGB threshold being the tax for lower luminosity values. For example, R = 25, G = 25, B = 25.
    In relation now to Figure 6, the execution of the motion detection procedure is shown in a partial functional block diagram.
     fifty
    At first, an analysis of the visual characteristics of the first aligned image F0 is performed in the non-RNP screen region and searches for it in the adjacent F1. This path is sequential, so that the characteristic found in the image n, Fn will be searched in the non-RNP region of the image n + 1, Fn + 1.
    Once the movement of all the characteristics of the non-RNP region has been analyzed and stored with a probability function in the second repository; for example, with a Gaussian function, the movement of all images in Cartesian axes is modeled. For the modeling of this movement, the maximum likelihood method MLE is used.
     5
    Once the movement of all the characteristics of the non-RNP region is modeled by the MLE method in the two axes of Cartesian coordinates, the characteristics of the image n are found in the potentially RPP screen region and then look for them in the image n +1.
     10
    With the movement of the characteristics of the potentially RPP screen region analyzed, the results are compared with the MLE model of the movement of the non-RNP screen region. To identify whether the movement of a characteristic point of the region potentially RPP screen corresponds to the movement of the multimedia content shown by screen 13, it is necessary to study the probability that this point has to fit its movement within the 15 models of the previous stage, referring to the region does not display RNP. The formula that measures this probability is defined as:

    Pt = Px * Py
     twenty
    Px being the probability that the movement on the horizontal axis x of the analyzed RPP point can be modeled by the parameters of the model of the previous stage for the movement of the non-RNP screen region on the x axis.

    Py being the probability that the movement in the vertical axis and of the point of the region 25 potentially analyzed RPP screen, can be modeled by the parameters of the model of the previous stage for the movement of the region no RNP screen in the y axis.

    If the probability obtained for each Pt point is less than a predetermined second Threshold2 threshold value, it is stored as a potentially PPP screen point in an image 30 buffer within the application server repository 15.

    If Pt> Threshold2 it is ruled out that this point is part of the multimedia content shown on screen 13.
     35
    Figure 7 shows the division of the region of interest ROI into four quadrants 1 to 4.

    In relation to Figure 8, the second phase of the motion detection procedure where each buffer image is divided is shown in a partial functional block diagram, resulting from storing the cloud of points obtained in the first phase of the detection procedure. of movement in four quadrants, figures 6 and 7. Each quadrant must be transformed to make the reference system the same for all of them.

    Arranged the buffer image correctly, the integral image I is computed; which accumulates 45 PPP points from the origin of coordinates to the corners for each quadrant. With the integral image calculated, the distance matrix D is computed which determines how far away a PPP point is from the center of the image.

    Once the previous relationships have been obtained, the R ratio matrix is obtained; being R = l / D, 50 whose maximum values offer, for each quadrant, the points that are estimated as the screen corner. As the maximum value for each quadrant reflects the point that has the greatest relationship between a greater number of PPP points and a smaller distance to the origin of coordinates.
    With these four defined points, one per quadrant, the rectangle that estimates the screen region is computed.

    In relation to Figure 9, the path followed in the procedure for detecting possible frames is shown in a functional block diagram where it is intended to obtain the quadrilateral that optimizes the surface occupied by the screen through the geometric study of the image. To do this, the histogram of oriented gradients of the HOG image on the potentially RPP screen region is calculated and divided into four quadrants.

    The objective of the previous procedure is to evaluate, for each quadrant, which is the point of the HOG map that has a greater probability of falling on a crosshead. This means, the point that is most likely to fall on a predominantly horizontal line and a predominantly vertical line at the same time. Well, the point with the highest cross probability will be the one with the highest probability of representing a corner of the screen. fifteen

    The first step is to obtain, for each point of the HOG quadrant, the probability that the point is on a horizontal line Ph and the probability that the point is on a vertical line Pv and from there obtain a Pcrucet matrix such as:
     twenty
    Pcruceta = Ph + Pv

    Being Ph for each point, the cumulative probability on its horizontal of being located in a straight line of horizontal direction.
     25
    Pv being for each point, the cumulative probability on its vertical of being located in a straight line of vertical direction.

    In a second step, we calculate for each quadrant the statistical correlation of the Pcruceta matrix with its adjacent quadrants. To do this, we add the maximum value, per row, of its adjacent horizontal matrix to the values of the matrix of each quadrant 30. And we add the maximum value, per column, of its adjacent vertical matrix.

    From this statistical correlation a correlated matrix Pc is obtained that strengthens the geometric coherence between the results of each quadrant, and favors that the points obtained as 35 potential corners in each quadrant are related to its adjacent quadrant.

    Once we have obtained the correlated matrix for each quadrant, the distance matrix D is computed. It determines how far each HOG point is from the center of the image. And it is operated to get a matrix relation R, being R =. 40 DPc

    From this matrix, the Z points of greatest value are taken for each quadrant. Each of these points represents the point that is most likely to be a corner of the screen.

    For each quadrant, the Z points with the highest value in the R matrix are grouped into M 45 subsets: grouping the points that are close and analyzing as a centroid the point with the highest value of the grouping in the R matrix. Joining the M centroids From each of the four quadrants, we estimate an N number of quadrilaterals, where N is equal to M4. This grouping procedure is performed to reduce the computation time; grouping by sets reduces the number of points to evaluate, since M4 is always less than or equal to 50 Z4, that is, M≤Z.

    At a later stage, the resulting quadrilaterals are reevaluated. Giving greater value to those who have their corners better aligned and to those who present a relationship -threesome- closer 16: 9. Well, this relationship is the most common for current screens.

    Once each quadrilateral is reevaluated, the N ’quadrilaterals 5 that are most likely to represent the ROI are selected from the total. From this stage N ’solutions per image emerge.

    Figure 10 shows the logical flow in functional block diagram of the path followed in the homogenization procedure, the purpose of this procedure being to complement the conclusions drawn from the previous stage, see Figure 9, offering a reevaluation of the N 'quadrilaterals obtained in each image. In the first phase, each aligned image is processed to detect characteristic borders. This procedure, which is carried out in grayscale, allows to study whether the lines of the quadrilateral coincide with an edge. Once this stage has been completed, the variance of the intensity of the image is analyzed along each line of the quadrilateral, potential screen edge, and the results are recorded.

    Using the above results as metrics, each quadrilateral is weighted and the N ’quadrilaterals are returned with the reassessed scores according to: 20

    - The value of the variance. The smaller the variance along the lines of the quadrilateral, the greater the reassessment of the same, as a screen frame usually has solid and homogeneous colors with little variance in intensity.
     25
    - The level of alignment between the detected edge and the quadrilateral lines. If the characteristic edges detected for the image coincide with the lines detected for the quadrilateral, the results of the reevaluation will improve for it. As the probability that a quadrilateral represents a screen frame increases if its lines coincide with characteristic edges detected in the image. 30

    Figure 11 shows the logical flow in a block diagram of the path followed in the process of unifying the results offered by the method of detection of possible frames, Figure 9, and of homogeneity, Figure 10, only those quadrilaterals containing the rectangle offered by the motion detection procedure, Figures 6 and 8, with a maximum error of a percentage P of the dimensions of the aligned image.

    Once this step is filtered, the score of each of the selected quadrilaterals is reevaluated, giving a higher score to those that are closer to the 40 rectangle offered by the motion detection procedure. Next, the quadrilateral with the highest score for each of the frames is stored.

    Now, for each image, you get the corner point of the rectangle in each quadrant. And among all the corners of each quadrant, those that are closest to the center of the image are selected. Finally, with the corners defined for each quadrant, we define the region of interest ROI.

    Figure 12 shows how the characteristics found for each region of interest ROI of the aligned images are treated depending on how far they are from the center of the region of interest ROI. The characteristics identified in the inner region of the image are taken in their entirety. And the characteristics that are in a perimeter region of the image are discarded according to what they approach the perimeter. This discarding trend in the outer perimeter of the region of interest ROI is linear and is performed
    so that for the edge of the image no characteristic is evaluated. The graph on the left shows, as a percentage, the number of characteristics evaluated in relation to the width of the ROI region of interest. The graph on the right shows, as a percentage, the number of characteristics evaluated in relation to the height of the ROI region of interest.
     5
    Figure 13 shows an example of a system that executes the method for capturing images from a camera 12 of a portable device 11 of an audiovisual screen 13 in which multimedia content of interest to a user of the portable device 11 is reproduced.
    10
    权利要求:
    Claims (22)
    [1]

    1. A method for capturing images from a camera (12) of a portable device (11) of an audiovisual screen (13) in which multimedia content of interest to a user of the portable device (11) is reproduced; characterized in that the method 5 comprises the steps of capturing, by means of at least one camera (12), of at least one source image (14) of the audiovisual screen (13); storage, in a first repository of the portable device (11), of the plurality of source images (14); transmission, through a first two-way communications interface, of the source images (14) to an application server; reception, by means of a second bi-directional communication interface 10 included in the application server, of the source images (14); sending the received images to a processor unit connectable to a second repository, which stores a set of multimedia contents where each stored multimedia content has associated at least one complementary multimedia content; conducting a match search, by means of the processing unit, between the set of images received and the multimedia contents stored in the second repository; provides a link to a complementary multimedia content associated with the stored multimedia content, by means of the processing unit, in case the result of the search procedure is affirmative; transmission of the link provided from the second communication interface to the first communication interface; and supply of the link to the user of the portable device (11) for display.

    [2]
    2. Method according to claim 1; characterized in that the capture of the source images is carried out continuously or discontinuously; being spaced in time at regular or irregular intervals. 25

    [3]
    3. Method according to claim 1; characterized in that the stored multimedia content has associated at least one complementary multimedia content.

    [4]
    4. Method according to claim 2; characterized in that the complementary multimedia content 30 is editable depending on the location of the user, the profile of the user and the timing of the broadcast of the source multimedia content.

    [5]
    5. Method according to claim 1; characterized in that the coincidence search stage comprises measuring the brightness level of the images received.

    [6]
    6. Method according to claim 5; characterized in that the matching search stage comprises resizing and aligning the received images with the first received image. 40

    [7]
    7. Method according to claim 6; characterized in that the matching search stage comprises the detection of the screen region (ROI) for the aligned images.

    [8]
    8. Method according to claim 6; characterized in that the coincidence search stage comprises a motion detection procedure based on obtaining a rectangle of maximum surface defined by the corner points.

    [9]
    9. Method according to claim 8; characterized in that the motion detection procedure defines the corner points of the maximum surface rectangle from a point cloud resulting from evaluating for each aligned image the movement of its characteristics or descriptors.
    [10]
    10. Method according to claim 9; characterized in that the detection of the corners, from among the whole cloud of points, coincides with those points that have greater value in the
    relationship obtained by dividing an integral image matrix (I) by a distance matrix (D) for each of the aligned images.

    [11]
    11. Method according to claim 10; characterized in that the corner detection stage divides the aligned image into four quadrants and identifies the corner as the maximum value, for each quadrant, of the ratio matrix obtained by dividing the integral image matrix (I) by the distance matrix ( D).

    [12]
    12. Method according to claim 6; characterized in that the match search stage comprises a method of detecting possible frames based on the geometric identification of elements similar to a screen frame, within each aligned image.

    [13]
    13. Method according to claim 12; characterized in that the geometric identification of elements similar to a screen frame is based on the evaluation of the results of a matrix that computes, for each point of the image, the cumulative probability of being a corner depending on the distance to the center of the image aligned.

    [14]
    14. Method according to claim 13; characterized in that the cumulative probability for each point of the image is defined from the evaluation of the 20 HOG orientations for each of the lines, horizontal and vertical, in which the point is located.

    [15]
    15. Method according to claim 14; characterized in that the matrix results are reassessed based on how aligned their sides are and how close they are to the 16: 9 ratio. 25

    [16]
    16. Method according to claim 6; characterized in that the coincidence search stage comprises a homogenization procedure that reinterprets the results obtained by the method of detecting possible frames.
     30
    [17]
    17. Method according to claim 16; characterized in that the result obtained by the homogenization procedure is positively reassessed if the color intensity along frame lines is homogeneous.

    [18]
    18. Method according to claim 16; characterized in that the result obtained for the homogenization procedure is positively reassessed if the detected frame line falls on an edge of the image.

    [19]
    19. Method according to claim 6; characterized in that the coincidence search stage comprises a unification procedure that allows obtaining a common and unified solution for the results of the motion detection procedure and the detection of possible frames and homogenization.

    [20]
    20. Method according to claim 19; characterized in that the selection of the single solution comprises obtaining a final quadrilateral that depends on the proximity of the results of the motion detection stage and possible frames and homogeneity.

    [21]
    21. Method according to claim 9; characterized in that the evaluation of the descriptors found in the region of interest (ROI) based on a linear discard of peripheral descriptors, depending on the distance of the descriptors to the periphery of the region of interest (ROI).

    [22]
    22. Method according to claim 1; characterized in that the search for coincidence includes the comparison between histograms of visual words as a function of the frequency of occurrence of similar visual words.
    类似技术:
    公开号 | 公开日 | 专利标题
    US9519968B2|2016-12-13|Calibrating visual sensors using homography operators
    US10372970B2|2019-08-06|Automatic scene calibration method for video analytics
    TWI485650B|2015-05-21|Method and arrangement for multi-camera calibration
    US10896518B2|2021-01-19|Image processing method, image processing apparatus and computer readable storage medium
    US9305206B2|2016-04-05|Method for enhancing depth maps
    US8331619B2|2012-12-11|Image processing apparatus and image processing method
    ES2390295T3|2012-11-08|Identification of people using multiple types of tickets
    JP2014241155A|2014-12-25|Method for providing descriptor as at least one feature of image, and method for matching feature
    US20150146032A1|2015-05-28|Light field processing method
    US20150055824A1|2015-02-26|Method of detecting a main subject in an image
    US20120092329A1|2012-04-19|Text-based 3d augmented reality
    WO2017092679A1|2017-06-08|Eyeball tracking method and apparatus, and device
    US10803365B2|2020-10-13|System and method for relocalization and scene recognition
    WO2016107638A1|2016-07-07|An image face processing method and apparatus
    Gade et al.2012|Occupancy analysis of sports arenas using thermal imaging
    Jabri2011|Camera calibration using court models for real-time augmenting soccer scenes
    JP2005031044A|2005-02-03|Three-dimensional error measuring device
    US7006706B2|2006-02-28|Imaging apparatuses, mosaic image compositing methods, video stitching methods and edgemap generation methods
    WO2018112790A1|2018-06-28|Image processing method and apparatus
    CN106713740A|2017-05-24|Positioning and tracking video shooting method and system
    ES2684690B1|2019-07-10|Method for capturing images from a portable device
    JP2013029996A|2013-02-07|Image processing device
    US8965106B1|2015-02-24|Detecting image inconsistencies
    KR101632069B1|2016-06-20|Method and apparatus for generating depth map using refracitve medium on binocular base
    CN112272292B|2021-06-29|Projection correction method, apparatus and storage medium
    同族专利:
    公开号 | 公开日
    ES2684690B1|2019-07-10|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    ES2227940T3|1994-10-25|2005-04-01|United Parcel Service Of America, Inc.|AUTOMATIC ELECTRONIC CAMERA TO RECEIVE A LABEL IMAGE.|
    ES2247659T3|1997-02-14|2006-03-01|Sony Corporation|METHOD AND CONVERSION DEVICE OF IMAGE SIGNAL.|
    US20120154633A1|2009-12-04|2012-06-21|Rodriguez Tony F|Linked Data Methods and Systems|
    US20130265451A1|2012-04-10|2013-10-10|Samsung Electronics Co., Ltd.|Apparatus and method for continuously taking a picture|
    ES2447640A1|2012-08-08|2014-03-12|Consejo Superior De Investigaciones Científicas |Method for transforming images into clouds of dots of multidimensional spaces, method for identifying objects and individuals, segmentation method, method for locating points of interest and uses thereof|
    US20150178786A1|2012-12-25|2015-06-25|Catharina A.J. Claessens|Pictollage: Image-Based Contextual Advertising Through Programmatically Composed Collages|
    法律状态:
    2018-10-04| BA2A| Patent application published|Ref document number: 2684690 Country of ref document: ES Kind code of ref document: A1 Effective date: 20181004 |
    2019-02-04| PC2A| Transfer of patent|Owner name: BIYECTIVA S.L. Effective date: 20190129 |
    2019-07-10| FG2A| Definitive protection|Ref document number: 2684690 Country of ref document: ES Kind code of ref document: B1 Effective date: 20190710 |
    2021-09-28| FD2A| Announcement of lapse in spain|Effective date: 20210928 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES201700449A|ES2684690B1|2017-03-30|2017-03-30|Method for capturing images from a portable device|ES201700449A| ES2684690B1|2017-03-30|2017-03-30|Method for capturing images from a portable device|
    [返回顶部]